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Abstract. We consider invariant Riemannian metrics on compact ho- 
O ' mogeneous spaces G/H where an intermediate subgroup K between G 

lj ' and H exists, so that the homogeneous space G/H is the total space 

of a Riemannian submersion. We study the question as to whether en- 
larging the fibers of the submersion by a constant scaling factor retains 
the nonnegative curvature in the case that the deformation starts at 
a normal homogeneous metric. We classify triples of groups (H, K, G) 

T^ ' where nonnegative curvature is maintained for small deformations, us- 

ing a criterion proved by Schwachhofer and Tapp. We obtain a complete 
classification in case the subgroup H has full rank and an almost corn- 
ed ' plete classification in the case of regular subgroups. 
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1. Introduction 

<N 

J> . The study of manifolds with nonnegative or positive sectional curvature 

is one of the classical fields of Riemannian geometry. Examples of mani- 
folds which admit metrics of strictly positive curvature are scarce, but if 
£SJ ', one relaxes the curvature condition to include manifolds of nonnegative cur- 

^ ' vature, the situation is different. For instance, any compact homogeneous 

space admits a metric of nonnegative sectional curvature. 

The class of manifolds supporting Riemannian metrics of nonnegative cur- 
vature is larger; nevertheless, only a few methods are known and it is also 
of interest to explore the family of all Riemannian metrics with nonnega- 
K> | tive curvature on a given manifold. Some results in this connection were 

recently obtained by Schwachhofer and Tapp |ST| in the setting of compact 
homogeneous spaces. They investigate certain deformations of a normal ho- 
mogeneous metric on a compact homogeneous space G/H within the class 
of G-invariant metrics. 

They prove the following structural result. The family of invariant met- 
rics is star-shaped with respect to any normal homogeneous metric if the 
symmetric matrices corresponding to invariant metrics are parametrized by 
their inverses. Thus the problem of determining all invariant metrics with 
nonnegative curvature reduces to determining how long nonnegative curva- 
ture is maintained when deforming along a linear path, starting at a normal 
homogeneous metric. 



Date: February 25, 2013. 

2000 Mathematics Subject Classification. Primary: 53C30; Secondary: 53C21; 57S15. 

1 



2 M. M. KERR AND A. KOLLROSS 

Schwachhofer and Tapp investigate this problem for the simplest nontriv- 
ial case, namely they assume there is an intermediate subgroup K between H 
and G and study metrics obtained through shrinking or enlarging the fibers 
of the Riemannian submersion G/H — > G/K by a constant scaling factor. 
As they point out, a metric obtained by shrinking the fibers can be inter- 
preted as a submersion metric obtained via a Cheeger deformation |Chj and 
hence shrinking the fibers always preserves nonnegative curvature. On the 
other hand, if one enlarges the fibers by a constant scaling factor, whether 
nonnegative curvature is maintained for small deformations depends on the 
triple (H, K, G) . In |ST] they find a criterion on the Lie algebra level, see 
Theorem l2.1l below. The preservation of nonnegative curvature under scaling 
up first appears in Grove and Ziller's paper on Milnor spheres |GZ| . 

This condition holds in particular if (K, H) is a symmetric pair, an ob- 
servation which yields a new class of examples for nonnegatively curved 
metrics. To study the Lie-theoretic condition found by Schwachhofer and 
Tapp is interesting in its own right and determining which triples (H, K, G) 
of compact Lie groups satisfy the criterion turns out to be an intriguing 
problem. In [KK] we classify all such triples in the special case where G 
is simple of dimension up to 15. In the present article, we use root space 
decompositions to study the problem for three classes of examples. 

The first class consists of triples (H, K, G) where G/K is a symmetric 
space with ik(G/K) = rk(G) and H arises as the intersection of K with 
a subgroup of maximal rank in G. The second class consists of all triples 
(H,K,G) for which rk(H) = rk(i^) = rk(G). For those two classes of 
spaces we prove that enlarging the fibers of the submersion G/H — > G/K 
maintains nonnegative curvature if and only if {K, H) is a symmetric pair. 
In the last part of this article we consider more generally the criterion for 
triples (H, K, G) where H, K are regular subgroups of G. Here there are 
also examples satisfying the criterion such that (K, H) is not a symmetric 
pair. Some of these examples were known before by work of [STj . but we 
also present a new class of examples, see Section [5j We obtain an almost 
complete classification in this case. 

It is a pleasure to thank the referee for helpful suggestions. 



2. Preliminaries 

Let H C K C G be compact Lie groups and let f),£,fl, be their respective 
Lie algebras. Let go be a biinvariant inner product on g. Let s be the 
complement of t in g and let m be the complement of f) in t such that 
= f)ffim©s is an orthogonal decomposition with respect to go . Set p = m©s. 
Then p can be identified with the tangent space of the homogeneous manifold 
G/H at the point 1H. The restriction of go to p defines an Ad#-invariant 
inner product and thus a G-invariant Riemannian metric on G/H, which we 
will also denote by go. 
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For any element X in p, we write X = X m + X s , where X m £ m denotes 
the vertical component and X s £ s denotes the horizontal component of X. 
We study G-invariant metrics on G/H which are obtained by deforming the 
Riemannian metric go on G/H such that the length of vectors which are 
tangent to fibers of the Riemannian submersion 

G/H -»■ G/K (2.1) 

is scaled by a constant and the length of vectors normal to the fibers is 
unchanged. That is, we consider the one-parameter family of Riemannian 
metrics gt on G/H, where t € (—00, 1), 

g t (X,Y):= Y ^- t -g (X m ,Y m )+g (X s ,Y s ). (2.2) 

It is well known that the normal homogeneous metric go has nonnegative 
curvature. As Schwachhofer and Tapp point out in |STj . a metric gt where 
t < can be reinterpreted as a submersion metric using Cheeger's con- 
struction |Ch] in the following way, see also [Z, Section 2]. Assume G is 
equipped with the biinvariant metric go and the homogeneous space K/H 
is equipped with the metric A 2 ■ <?olt x r Then an isometric action of K on 
G x K/H is defined by k ■ (g,aH) = (gk^ 1 ,kaH). The quotient by this 
action is diffeomorphic to G/H and the submersion metric corresponds to 
(|2.2p with t = — 1/A 2 . Since the metric on the total space of this submersion 
has nonnegative curvature, it follows from O'Neill's formula that the metric 
on the base space has nonnegative curvature. 

However, metrics gt with t > do not have an such an interpretation and 
whether they have nonnegative curvature depends on the triple (H,K,G). 
Schwachhofer and Tapp prove the following condition on the Lie algebra 
level. 



Theorem 2.1. [ST] 

(1) The metric gt has nonnegative curvature for small t > if and only 
if there exists some C > such that for all X and Y in p, 

\[X m ,Y m ] m \<C\[X,Y}\. (2.3) 

(2) In particular, if (K, H) is a symmetric pair, then gt has nonnegative 
curvature for small t > 0, and in fact for all t G (—00, 1/4]. 

The first part of (2) follows immediately from the observation that con- 
dition (|2.3h always holds when [m, m] C fj, in which case the left hand side 
of the inequality is obviously zero for all X and Y in p. In particular, (|2.3p 
holds if J is abelian. 

When H is trivial the submersion (12. ip becomes K — > G — > G/K and gt 
is in fact a left-invariant metric on G. In this case, Schwachhofer [S] proved 
gt has nonnegative curvature for small t > only if the semisimple part of 
t is an ideal of q. In particular, when rj is simple and t is nonabelian, one 
does not get nonnegative curvature by enlarging the fibers. 
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Here, a pair (K, H) of compact Lie groups H C K is said to be a symmet- 
ric pair if there exists an automorphism a of the Lie algebra t with <r 2 = idt 
whose fixed point set is f). In this case we also say that (t, rj) is a symmetric 
pair (of Lie algebras). 

It is shown in [ST] that a number of examples (H, K, G) satisfy the hy- 
pothesis of Theorem 12. 11 Among them are the following chains H C K C G, 
where [m,m] <2 f): 

• SU(3) C SU(4) ^ Spin(6) C Spin(7), 

• G2 C Spin(7) C Spin(p + 8), where p G {0, 1}, and 

• SU(2) C SO(4) C G 2 , where SU(2) is contained in SU(3) C G 2 . 

Notation: We will denote the Lie algebra of the compact Lie group of 
exceptional type G 2 by Lie(G 2 ) in order to avoid confusion with the notation 
01; 02, ■■■ j 0m in Theorem 16. 5| by which we denote the simple ideals of a Lie 
algebra g. 



3. A SPECIAL CLASS OF LlE GROUP TRIPLES 

In what follows, we will review some general facts about the structure of 
real simple Lie algebras. Let T be a maximal torus of G and let t be its 
corresponding Lie subalgebra. Consider the adjoint representation Ad: G — > 
Aut(g) of G restricted to T. As a T- module, the Lie algebra g decomposes 
into the trivial module t and its go-orthogonal complement t , which further 
decomposes into a sum of pairwise inequivalent two-dimensional irreducible 
representations g„ of T such that for each element H € t, Ad(exp(ff)) is 
the action on g„ by a rotation of the form 

cos(a(H)) -sm(a(H))\ 
sm(a(H)) cos(a(H))) 

with respect to a suitable basis. Via Ad(exp(ff)) = exp(ad#) we pass to 
the Lie algebra level; for each g£r, we may choose a go-orthonormal basis 
(X a ,Y a ) such that 

ad^PQ = a(H)Y a , ad H {Y a ) = -a(H)X a . 

Note that g^ = jj?" a , in fact, replacing the ordered basis (X a ,Y a ) by (Y a ,X a ) 
reverses the rotation. Apart from this ambiguity, a is uniquely determined 
by the invariant subspace g^. We refer to the spaces g^ as the real root 
spaces of g. Let R be the set of nonzero elements a € t* such that there 
exists a nontrivial T-module g„ in t . Now choose a vector v G t such that 
a(v) 7^ for all a G R. Let R + := {a € R \ a(v) > 0}. We say R + is the set 
of positive roots of g and R = R + U — R + is the set of roots. Let g c = g (£> C 
denote the complexified Lie algebra. For any a € t* one defines 

C = [X G g c I ad H (X) = ia(H)X for all H G t} . 
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Whenever g^ ^ 0, we say g^ is a root space of g. For each a E R+, define 
the following elements of g c : 

±L a — A a lien F-'—a — -^-a ~r 1±ai 

so that we have 3Ajj{E± a ) = ia{H)E± a . In this way we obtain the root 
space decomposition 

g c = (t®c)e0 



3a • 



Of course g„ ^ if and only if a E -R. In this case g„ = CE a . 

Remark 3.1. For each a, the basis vectors X a ,Y a are determined only up to 
rotation: for any t E R we may replace E? Q and E_ a by e 4t S Q and e~ lt E_ a , 
which amounts to replacing X a and Y a by 

cos(t)X Q + sin(t)Y" a and — sin(t)X a + cos (t)Y a . (3-1) 

It follows from the Jacobi identity that [0^,0?] C g„ + « for all a,/3 E t*. 
In particular, if a + j3 is not a root, [0^,0?] = 0. On the other hand, we 
have [g£,gjp] = g^ +/3 whenever a + /3 / [H, Thru. 4.3, Ch. III]. We define 
the number N a> p by [E a ,Ep] = N a> pE a+ p if a, f3, a + /3 E R. 

Let r be a subset of R. Then T is said to be symmetric if whenever 
a E r, then —a E T. The set T is said to be closed if whenever a^GT and 
a + ft E i?, then a + /3 E T. 

We will now consider examples which arise from a special class of com- 
pact symmetric spaces G/K, namely those for which xk(G/K) = rk(G), i.e. 
s contains a maximal abelian subalgebra of g. The simply connected irre- 
ducible spaces of this type are SU(n)/SO(n), SO(2n+l)/SO(n+l) x SO(n), 
SO(2n)/SO(n) x SO(n), Sp(n)/U(n), E 6 /Sp(4), E 7 /SU(8), E 8 /SO'(16), 
F 4 /Sp(3)Sp(l), G 2 /SO(4), cf. [HI Table V, Ch.X]. For the spaces in this 
class, the Satake diagram of G/K is the same as the Dynkin diagram of G, 
but with uniform multiplicity one. The corresponding involution a on g 
induces an involution of g which acts as minus identity on t © C and 
sends each root to its negative. Furthermore, every real root space g^ is 
u-invariant, with a one-dimensional fixed point set. For each basis (X a ,Y a ) 
we have 

a(X a ) = X a , a{Y a ) = -Y a (3.2) 

after an appropriate rotation as in (|3.ip . In particular, for the subalgebra £, 
t = span{X Q | a E R+} and for its complement, s = tffispan{Y^, | a E R+}- 
Assume f) is spanned by a subset of {X a \ a E i?+} and (K,H) is not 
a symmetric pair. Then the go-orthogonal complement m of f) in t con- 
tains two elements of the form X\, X^ such that the m-component of their 
bracket LYA,X M ] m is nonzero (this implies A + \x / 0). Since [X\,X^] E 
gA+^©gA-/i©g^-A©g-A-/x, we may assume (possibly after interchanging A 
and /jl) that at least one of A ± \x is a positive root of g for which X\± fJL E m. 
Let v = A ± fj,, where the sign is chosen such that v E i?+ and X v E m. 
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The set of roots which appear as nonzero linear combinations of A and li 
with integral coefficients forms an irreducible rank two root system R(X, li) 
which is of type A2 or B2. (No root system of type G2 ever occurs as a 
proper subset of an irreducible root system.) 

We will now show that we may choose two linearly independent elements 
a, f3 € {A, fi, is} such that a — (3 € R+ and neither a + (3 nor 2a — (3 is a root. 
In case R(X, ll) is of type A2, choose a := A, /? := li if v = A — fx and choose 
a := is, (3 := fx if is = A + fx. Assume R(X,li) is of type B2 and is = A + fx. 
Then either A and li are orthogonal short roots, in which case we set a := is, 
(3 := ll, or A and \x are of different length and enclose an angle of ^p, in 
which case we choose a := is and (3 € {A, \x} to be the long root. 

Now assume R(X, li) is of type B2 and is = A — ll. Then either A and li 
are orthogonal short roots, in which case we set a := A, (3 := is, or A and 
li are of different length and enclose an angle of j, in which case we choose 
a G {A, ll} to be the long root and j3 := v. 

For any H € t and for any real constant i], define X = X a + H and 
Y = Xp + 7]Y a _p. We show that [X m , Y m ] / 0, and [X, Y] = 0. We recall, 

[H,Xp] = (3(H)Yp, [H,Y a _p] = -(a(H) - (3{H))X a _p. 

Furthermore, using the fact that a + f3 is not a root of q, we obtain 

4[-Xq,, Xp] = [E a + E- a , Ep + E_p] = N a _pE a _p + N_ a ^Ep_ a 
= N a _p(X a _p - iY a ^p) + N_ a> p(X a _p + iY a _p). 

It follows that N a _p = N- a> p € R \ {0} and we have shown that 
[X m ,Y m ] = [X a ,Xp] = ±N a ,_pX a _p. 

Similarly, we calculate, using the fact that 2a — (3 is not a root of 0, 

&[X a ,Y a _p] = i[E a + E_ a ,E a _p - Ep_ a ] = -iN a> p_ a Ep + iN_ a ^ a _pE_p 
= -N a ,p_ a {iXp + Yp) + N_ a , a _p{iXp - Yp). 

Since [X a , Y a -p] € s it follows that N a ^p_ a = N_ a ^ a _p G R \ {0}. We have 
shown 

1 
[X a ,Y a -p\ = —-N ay p- a Yp. 

Thus we have 

[X,Y] = [X a + H,Xp + 7 1 Y a _p} = 

= ^N a) _pX a _p - ^vN a ,p- a Yp + P(H)Yp - r)(a(H) - f3(H))X a _p. 

Since a and j3 are linearly independent, there exists an element H G t which 
solves the equation 

\N at _pN at p_ a + 2f3(H)(a(H) - (3(H)) = 0. 
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Having chosen such an H G t, we set 77 := — 2f3(H)/N a ^^ a , then 

[X,Y]=0 

and, on the other hand, [X m ,Y m ] m = [X a ,Xp] m / 0. This proves that 
condition (|2.3p does not hold. 

Using Borel-de-Siebenthal theory [Ol §3], we can describe the triples 
(H, K, G) covered by the above calculations more explicitly. Let us as- 
sume we have chosen bases X a , Y a of the real root spaces of q as described 
above, i.e. such that t is spanned by {^ a |a G R+} and rj is spanned by 
{X a |a G S+} for some subset 5+ C R+. Let I be the linear subspace of q 
spanned by tU {^ a |a G S+} U {l^|a G 64.}. Consider the complexification 
l c C C . Then ( c = (i®C)® Qe s0a > wh ere 5 := 5+ U -S+. By P 
Prop. 15, 1 §3], I is a subalgebra of q if and only if the subset S is closed 
and symmetric. In our case, 5 is symmetric by definition. Let us show that 
it is also closed: Let a, (3 G S such that a + /3 G R. Following [Wa] . we set 
\a\ := a if a G i?+ and \a\ '■= —a if —a G -R+. Furthermore, we are using 
the convention that whenever a + /3 ^ R we set 7V a 3 = 0, E a+ p = and 
whenever a G" i?, we set X a = 0. We compute 

4[-Xq,, Xp] = [E a + -E-a, Ep + ^-^j 

= {X a) pE a+ p + N_ a _pE_p^ a ) 

+ {X a -pE a _p + N_ a) pE_ a+ p) 
= 1N a) pX\ a+ p\ + 2A r ai _ /3 X| Q ,_ /3 |. 

Since -/V Q/ g 7^ and rj is spanned by {X a |a G S+}, this shows that X\ a+ p\ G f) 
and hence a + /3 G 5. Thus I is actually a subalgebra of q and f) = ( n B. 
Moreover, rk(() = rk(g) and hence the corresponding subgroup L is closed. 
Conversely, let L C G be a closed subgroup containing a maximal torus T 
of G. Let cr: G — >■ G be an involution as described above and assume we 
have chosen a root space decomposition and vectors X a , Y a , a G R+ such 
that (|3.2p holds. Let X be the fixed point set of a and let H := K P\ L. 
Then the triple H C K C G is such that f) is spanned by X a , a G S+, for 
some subset S + C i? + . We have proved the following. 

Theorem 3.2. Assume (G,K) is a symmetric pair such that rk(G/K) = 
rk(G) and let a: G —> G be the corresponding involution. Let L be a ex- 
invariant subgroup with rk(L) = rk(Gr), and let H := K DL. Then the triple 
(H, K, G) satisfies condition jl2.3\) if and only if (K, H) is a symmetric pair. 

It is interesting to note that while condition (J2.3I) fails for the triples 
H C K C G where (K, H) is not a symmetric pair, it always holds for the 
triples H C L C G, since (L, H) is a symmetric pair. Indeed, the involution 
a leaves L invariant and thus induces an involution of the same kind as a 
(mapping each root to its negative and acting as minus identity on t) on L. 

We do not give a full list of the triples where Theorem 13.21 applies, but 
just illustrate the result by the following examples. 
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Corollary 3.3. The following chains H C K C G of compact Lie groups 
do not satisfy condition Ii2.3\) in Theorem \2. 1\ (1): 

(1) SO(ni)xSO(n 2 )xSO(n 3 ) c SO(ra) c SU(n), iy, > 1, ni+n 2 +n 3 = 
n. 

(2) [SO(ni + 1) x SO(n 2 ) x SO(n 3 )] x [SO(ni) x SO(n 2 ) x SO(n 3 )] 

C SO(n + 1) x SO(n) c SO(2n + 1), m > I, m + n 2 + n 3 = n. 

(3) U(ni) x U(n 2 ) x U(n 3 ) C U(n) C Sp(n), n^ > 1, ni +n 2 + n 3 = n 

(4) [SO(ni) x SO(n 2 ) x SO(n 3 )] x [SO(m) x SO(n 2 ) x SO(n 3 )] 

C SO(n) x SO(n) C SO(2n), where rii > 1, n\ + rt 2 + n 3 = n. 

(5) SO(3) ■ SO(3) • SO(3) C Sp(4) C E 6 . 

(6) SO(3)-SO(6) C SU(8)/{±1} C E 7 . 

(7) SO(3)-Sp(4) C SO'(16) C E 8 . 

(8) SO(3)-SO(3) C Sp(3)-Sp(l) C F 4 . 

Proof. All examples are constructed in the following manner. Choose a 
subgroup LcGof full rank such that (G, L) is not a symmetric paho Then 
determine the subgroup H C L (unique up to conjugacy) such that (L, H) 
is a symmetric pair satisfying rk(L/H) = rk(L). Here are the subgroups L 
chosen in the examples above: (1) S(U(ni)xU(n 2 )xU(n 3 )); (2) S0(2rii+l)x 
SO(2n 2 ) x SO(2n 3 ); (3) Sp(m) x Sp(n 2 ) x Sp(n 3 ); (4) S0(2m) x SO(2n 2 ) x 
SO(2n 3 ); (5) SU(3) • SU(3) • SU(3); (6) SU(3) • SU(6); (7) SU(3) • E 6 ; (8) 
SU(3) • SU(3). See [Ol Thm. 16, §3] for regular subgroups. □ 

4. Subgroups of full rank 

In this section, we consider the case of closed subgroups H C K C G of a 
simple compact Lie group G such that rk(iJ) = rk(i^) = rk(G). We show in 
Theorem 14.41 that in this case the triple (H,K,G) satisfies condition (|2.3p if 
and only if (K,H) is a symmetric pair. We will show in Theorem 14.41 below 
that we may restrict ourselves to chains H C K C G with rk(G) < 3. In 
the following lemma, we prove our result in this special case. 

Lemma 4.1. For the following chains of compact Lie groups H C K C G 
there exist elements X, Y G p such that [X,Y] = and [A" m ,Y m ] m ^ 0. 

(1) T 3 C S(U(3) x U(l)) C SU(4), 

(2) U(2)xSO(2) c S0(6) C S0(7), 

(3) U(2)xSO(2) c SO(5)xSO(2) c S0(7), 

(4) U(l)xU(l)xU(l) C U(3) C Sp(3), 

(5a) Sp(l)xU(l)xSp(l) C Sp(2)xSp(l) C Sp(3), 
(5b) Sp(l)xU(l)xU(l) C Sp(2)xU(l) C Sp(3), 
(6) T 2 c SU(3) C G 2 . 

Here T n denotes an n-dimensional torus. 



If (G, L) is a symmetric pair, then (L, H) will be a symmetric pair as well, since the 
involution corresponding to (G, L) commutes with a. 
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Proof. Case (4) is a special case of part (3) in Corollary 13.31 For all other 
cases we exhibit our vectors X, Y using explicit matrix representations. 

We identify so(n) with the set of skew-symmetric real nxn-matrices, u(n) 
with the set of Hermitian complex nxn-matrices, sp(n) with the set of Her- 
mitian quaternionic nxn-matrices. Let E UfJj denote the skew-symmetric ma- 
trix with the entry +1 in position {y, jj), the entry — 1 in position (/i, v) and 
zeros elsewhere, while F V(l denotes the symmetric matrix with the entry 1 
in positions (/i, v) and (u, fx) and zeros elsewhere. 

(1) Let fj = i 3 = {dlag(iti,it 2 ,it 3 , -i(h + t 2 + t 3 )) \ h,t 2 , t 3 G R}, 



f)Ct ' ' A 



A eu(3),z = -trA 

We take X = E 12 + E u and Y = E 23 + E u . Note that E 12 , E 23 G m 
while E u , E u G s. Then [X m , Y m ] = E 13 G m and [X, Y] = 0. 

(2) Here 

u(2) = {(£ "/) |F = Y<}cso(4), 

and in this way, f) = u(2)ffiso(2) C so (4) ©so (2) C i = 50 (6). Thus 
m = mi©m2 where mi = so(4)eu(2) andrri2 = so (6)0 (so (4) ©so (2)). 
We take X = E\§ — Eu, Y = E 25 + E 27 , where Ei 5 ,E 25 G tu and 
E\-j,E 2 -j G s. Then [X m ,y m ] = —E\ 2 has a nonzero m component, 
while [X, Y] = 0. 

(3) We have f) = u(2) ©so(2) C so(4) ffiso(2) C so(5) ©so(2), and thus 
m = mi©m2 where mi = so (5) ©so (4), and rri2 = so (4) © u(2). We 
take X = E 15 + E 16 and Y = E 25 - E 26 , where £15, E 25 G mi, 
and E\6, E 2 § G s. Then [X m ,y m ] = —E\ 2 , which has a nonzero 
m-component (in 1TI2), and [X, Y] = 0. 

(4) See Corollary E3 (3). 

(5) In both cases (5a) and (5b), we have the same m = mi©rri2 where 
mi = sp(2) © (sp(l)ffisp(l)) and m 2 = sp(l) © u(l). We may take 
X m = iF 12 and Y m = kF 12 (both in m 2 ). We take X s = E 23 + iF 13 
and Y s = jF 23 - kF 13 (in both cases). Then [X m ,Y m ] = -2j(F u - 
F 22 ), so that [X m , Y m ] m = 2jF 22 / 0, while [X, Y] = 0. 

(6) See [KKl Subsection 2.4 (lc)]. 

n 

Remark 4.2. Let H C K C G be a triple of compact Lie groups and suppose 
H' C K 1 C G' is another triple of compact Lie groups with G' C G such 
that for the orthogonal complement p' of \]' in g' we have p' C p, while 
for the orthogonal complement m' of f)' in fi' we have m' C m. Then it is 
sufficient to exhibit a pair of vectors X, Y G p' such that [X, Y] = but 
[X m , Y m ] m ^ in order to show that the triple (H, K, G) does not satisfy 
condition (jlT5)> : Since X m = X m ' , Y m = Y m ' , and [X m ' ,Y m '] G g', we know 
[X m ',Y m '] m ' = [X m ,Y" 



^mim 
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Remark 4.3. Conversely, with the notation as in the remark above, if the 
triple (H, K, G) satisfies condition (|2.3p then the triple (H 1 , K', G') also sat- 
isfies condition (I2.3p . In particular, if the group H is enlarged to H', condi- 
tion (|2.3p is preserved. 

In the following, we will consider the case of compact Lie groups H C 
K C G where H and K are closed subgroups of full rank in G such that 
(K, H) is not a symmetric pair. We may chose a maximal torus T of H , 
which is then also maximal torus of K and G, and consider a root space 
decomposition with respect to T. Using the notation of Section [3j there are 
subsets Rh C Rk C i?, such that 

f, c = t c 0QI i c = t c e 0Q , 

a£Rn aGRpc 

where fy c ,6 c , t c ,m c ,s c denote the complexifications of fy,£,m,s, t, respec- 
tively. Let R m = Rk \ Rh and R s = R\ Rk ■ The scalar product on t* is 
induced from g$. The root systems Rh and Rk of H and K, respectively, 
are symmetric, hence the subsets R m and R s are also symmetric, see (Oj 
Ch.l,§3.11]. 

Theorem 4.4. Let G be a simple compact Lie group and let H C K C G 

6e closed subgroups. If rk(H) = rk(if) = rk(G) i/ien either (K, H) is a 
symmetric pair or there exist elements X,Y £ p such that [X, Y] = and 

LY m ,y m ] m /o. 

Proof. Since (X, i?) is not a symmetric pair, there exist A,/i£ i? m such that 
[flAj0/x] *£- § C - Hence, ^ Q\+n C m c . Let R(\, //) denote the subset of i? 
consisting of all roots which are nonzero linear combinations of A and /i with 
integer coefficients. Because R(X, fj,) contains the six roots g±\, Q±^, 0±(a+^)> 
it is an irreducible root system of rank two. 

Since q is simple, i c acts effectively on s c and there is a root v € R s such 
that [0A>0y] 7^ 0. Let R(X,/j,,v) be the set of all roots which are nonzero 
linear combinations of A, /i, v with integer coefficients. Then R(X, /i, v) is a 
closed, symmetric subsystem of R and hence a root system of rank two or 
three, which contains R(X, /i) as a closed, symmetric, proper subsystem. 

We know A + v € R, thus the root system R(X, /x, v) is irreducible. Hence 
the inclusion -R(A, /i) C R(X, /j,, v) is of one of the following types 

(1) A 2 CA 3 , (2) A 2 CB 3 , (3) B 2 C B 3 , . . 

(4) A 2 cC 3 , (5) B 2 cC 3 , (6) A 2 cG 2 . { ^ L> 

(Maximal closed symmetric subsets of irreducible root systems are given in 
P Ch.l, §3.11].) 

Let q' be the semisimple part of the subalgebra of q spanned by the real 
root spaces q^, a G R(X, /j, v) and t. Then g' is in fact a simple Lie algebra 
isomorphic to su(4), so(7), sp(3), or Lie(G 2 ). Let f)' := \)C\q' and t' := IHq'. 
Then m' :=mflg' is the orthogonal complement of f)' in t and s' := s n q' is 
the orthogonal complement of t 1 in q'. In particular, Remark 14.21 applies. 
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We complete the proof by showing that for each type of inclusion R{X, /u) C 
R{X,fi,v) as enumerated in (|4.ip . the triple f)' C t' C g' corresponds to one 
of the triples in Lemma 14.11 Thus there exist X, Y € p' C p such that 
[X, Y}=0 and [X m ' , y m '] m ' = [X m , Y m ] m + 0. 

Since g v C s we have that t' C g' and since 0±A,0±/i>0±(A+^) C m, we 
know [m'jm'] % \). In particular m' / 0. Let i* be the subalgebra of g 
spanned by t and the real root spaces g^, a € R{X,fi). Since A,/x € R m , it 
follows that £* is contained in £'. 

When the subalgebra $* C g' is maximal, it follows that t* = £'. This is 
the case for all possible inclusions t* C g' enumerated in (|4.ip except (2) 
and (5). In case (2) we have A2 C A3 C B3, thus we further distinguish the 
cases (2a) t' = so (6) and (2b) t' = t* = u(3). In case (5) we distinguish the 
cases (5a) t' ^ sp{2) ®sp(l) and (5b) ¥ = t* = sp(2) ©u(l). 

In the cases (1), (4) and (6) where the semisimple part of t' is of type A2, 
it follows that fj' is abelian, as all six roots of R{X, fi) are contained in R m ; 
hence the triple r/ C £' C g' is determined, up to an automorphism of g'. 
We have one of the chains of Lie groups in Lemma |4.1| (1), (4), or (6). 

If the semisimple part of t' is of type B2 as in cases (3) and (5), then 
either six or eight roots of R(X, n) are contained in R m and it follows that f) 
is either abelian or its semisimple part is of type Ai. Note that there are two 
possibilities for the inclusion Ai C E>2, corresponding to the inclusions of Lie 
groups SU(2) C SO(5) and SO(3) C SO(5); in the first case the triple tj' C 
t' C g' corresponds to the chains (3) or (5) in Lemma [4. 11 In the second case, 
the triple [)' C !' C g' would correspond to either SO(3) x SO(2) x SO(2) C 
SO(5) x SO(2) C SO(7) in case (3) or U(2) x Sp(l) C Sp(2) x Sp(l) C Sp(3) 
in case (5). But in both cases (K,H) is a symmetric pair, a contradiction. 

In case (2a) we have g' = so (7) and t' = so (6). Hence ty C t' is a subalge- 
bra of rank 3 such that (£', \)') is not a symmetric pair. The only possibilities 
are r/ = u(2) ®u(l) and t)' = so(2) ®so(2) ®so(2). The first case is covered 
by Lemma 14.11 (2). the second case by Remark 14.21 and Lemma 14.11 (2). 

Finally, in case (2b) we have that t' = u(3), it follows that f) is abelian, 
since m' is at least 6-dimensional. Hence the triple r/ C t' C g' corresponds 
to a triple T 3 C U(3) C Spin(7). Using Remark 14.21 once more to replace 
Spin(7) by Spin(6), we see that this case is covered by Lemma I4TT1 (1) via 
the isomorphism SU(4) = Spin(6). 

We have now shown that in each case there exist elements X, Y G p such 
that [X, Y} = and [X m , Y m } m / 0. □ 

Corollary 4.5. Let G be a compact Lie group and let H C K C G be closed 
subgroups. Ifrk(H) = rk(K) = rk(G) then the triple satisfies condition A2.3\) 
if and only if for each simple factor Gi of G, at least one of the following 
holds. 

(i) {Gi n K, Gi fl H) is a symmetric pair. 
(ii) ftCt. 
(iii) gin!Crj. 
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In particular, if there is a simple factor Gi of G for which none of the above 
conditions holds, then there exist elements X, Y G p such that [X, Y] = 
and[X m ,Y m ] m ^0. 

Proof. Let g = 3 © 0i © . . . © m , where 3 is the center of g and where the 
Qi are the simple factors of g. Since ik(H) = rk(if) = rk(G), we have that 
I = 3 © (fli H l) © . . . © (fl m PI I) and F) = 3 © (fli n \)\ I © . . . © (fl m n &). Prom 
this fact it is obvious that condition fj2.3[) holds if and only if it holds for 
each triple (H n Gi,K n Gj, Gj) where i = 1, . . . , m. Now the first part of 
the corollary follows from Theorem 14.41 

Assume there is a simple factor Gi of G for which none of (i), (ii), (iii) 
above holds. Apply Remark 14.21 to the chain H n Gi C if n Gi C Gj to see 
that the second part of the assertion follows. □ 



5. New examples 

Theorem 5.1. The chain SU(n) C S0(2n) C S0(2n + 1), n > 2 satisfies 
condition h2. 



Proof. Let n > 2, g = so(2n + 1) and t = so(2n). We introduce a complex 
structure on R 2n by 



J 






-1 








1 




















-1 








1 






\ 



V 



-1 

1 



(5.1) 



Then u(n) = {A G so(2n) | J A = AJ}, the subset of all matrices in so(2n) 
which commute with J. The orthogonal complement mo of u(n) in so(2n) is 
then mo = {M G so(2n) | JMJ = M}, the subset of all matrices in so(2n) 
which anticommute with J. The one-dimensional ideal which is the center 
of u(n) is RJ =: 3, that is, u(n) = su(n) © 3. The orthogonal complement 
of \] in t is m = mo © 3. For the orthogonal complement 5 of tin g, we have 
[s,s] Ct 

Let A m , y m e m and A s , y 5 G s. Define X := X m + X s , Y := Y m + Y s 
in p = m©s. Let W G so(2n) be a matrix of rank two and let x G R 2n be a 
unit vector such that x G (kerVF)- 1 . Let A := |W(x)| and let y := W{x)/\. 
Since W is skew symmetric, we have im(VF) = (kerW^)- 1 - and W(v) _!_ v for 
all v G R 2ra . Hence (hevW)^ is spanned by the orthonormal vectors x and 
y. It follows that 

W = X(yx l - xy 1 ). 

Define S(x,y) := yx l — xy l for x,y G R 2n . We have seen that any rank two 
matrix in so(2n) is given by S(x,y) for some x,y G R 2n . In particular, we 
have [X s , Y B ] = S(x,y) for some x,y G R 2 ™. 
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For t G R we may define the pair (X, Y) G p x p by 

X := cos(t)X + sin(t)Y, Y := cos(t)Y - sm(t)X. 

Then we have [X,Y] = [X,Y] and [X m ,Y m ] m = [X m ,Y m ] m . In particular, 
by choosing t suitably, we have Y l = cos(t)y 3 — sin(i)X 3 = 0. Dropping the 
tildes, we may therefore assume that Y i = and hence Y m = Y m ° . 

Recall that SO(2n)/ U(n) is a symmetric space of rank r := |_§J, see [H] . 
Indeed, conjugation by the matrix J defines an involutive automorphism 
of t = so(2n) whose (+l)-eigenspace is f) ©3 and whose (— l)-eigenspace is 
mo- Let 



D 



Consider the maximal abelian subalgebra a C mo defined as follows. For 
n even, define a to be the space of all skew symmetric real 2n x 2n-matrices 
of the form 

ft x D \ 

t 2 D 

(5-2) 

\ t r Dj 

where £1, . . . , t r € R. If n is odd, define a to be the space of all skew 
symmetric real 2n x 2n-matrices of the form 



/ 


1 


°\ 








-1 


-1 








{0 1 





°/ 



/ hD 



t 2 D 



t r D 



V 



\ 







(5.3) 



where t\, . . . ,t r £ R. Since the isometric action of the group H on q given by 
restriction of the adjoint representation of G leaves mo, 3 and s invariant, we 
may replace (X, Y) by (Adf l (X),Adf l (Y)) for any h G H without limitation 
of generality, since we have 

|[Ad ft pO,Ad h (y)]| = \Ad h ([X,Y})\ = \[X,Y]\ 

and 

\[Ad h (X) m ,Ad h (Y) m ] m \ = |[Ad h (X m ),Ad^(y m )] m | 

= \Ad h ([X m ,Y m ]) m \ 
= \[X m ,Y m ] m \. 

It follows from the theory of symmetric spaces [H] that the subspace a 
intersects all orbits of the if -action on mo, in particular, there is an element 
h G H such that Adh(Y m ) G a and we may henceforth assume Y m G a, i.e. 
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Y m is of the form (J5.2H or (|5.3f) . We define a subalgebra of t isomorphic to 
r ■ so (4) as follows. If n is even, define 



[M 



fi 



An 



If n is odd, define 



\ 



A r ) 



£50(4r) 



Ai,...,A r €so(4) 



> . 



f 

K 


Mi 


A 2 


A r 




V 











G so(4r + 2) 



A 1 ,...,A r eso(4) 



> . 



Let P: t — > t\ be the orthogonal projection from t onto t\. Note that 
X' 3 , Y m E t\. Furthermore, the action of t\ on t leaves i-i and its orthogonal 
complement invariant, hence we have P([V, Y m ]) = [P(V), Y m ] for all V S t. 
We have 

\[x,y}\ > \[x,y] 1 \ > |p([x,y]*)| = \p([x m ,Y m ] + [x s ,y s })\ = 

= |p([x 3 ,y m ] + [X m °,Y m ] + [X S ,Y S ])\ = 

= \[X~>,Y m ] + [P(X m °),Y m ] + P([X S ,Y S })\ (5.4) 

Now let Pi, . . . , P r : t — > so (4) be the orthogonal projections onto the direct 
summands of t\ isomorphic to so (4) such that P^+i maps a matrix A = 
(ciij) £ {? to its 4 x 4-submatrix 



«4£+l,4£+l 



^4^+1, 4£+4 



,04^+4,4^+1 



04£+4, 4£+4 



Let J\ = Pi (J) = . . . = P r (J) be the complex structure on R 4 defined by 
(15.11) in the case n = 2. From (15.41) we have 



(5.5) 



[x,y] 1 > ]T|[ c Ji,p,(r m )] + [p,(x m »),p,(y m )] + p,([x*,y s 



v=i 



Since P„([X B ,y B ]) is a matrix of rank two or zero, we have P,([X S , Y ]) = 
[U S ',V S '\ = S(x,y) for two vectors x = (x 1 ,x 2 ,x 3 ,x 4: ), y = (2/1,2/2,2/3,2/4) G 
R 4 . We claim that there is a constant C > such that 



m -i / ^mimt2 



[x,y] f >c^-|[x m ,y m ] 
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for all X, Y G p. Each of the r summands on the right hand side in (15, 5p is 
of the form 

[W', v m '] + [u<, v m '\ + [u s \ v s '] 2 = [U, V} 1 

where U, V G p' = m' + s' is a pair of vectors for the triple 

(#',#',<?') = (SU(2),SO(4),SO(5)) 

where q' = I' © s' and t' = \]' © m' are orthogonal decompositions and 
where 3 is spanned by J\. Hence it suffices to verify the claim for the case 
n = 2. We have U i = cJ\ and V m = tD for some r G R. Consequently 
[Ui',V m '] = tE, where 

/ l\ 
10 
: ~ 0-100 

\-l 000/ 

and t = 2cr. Furthermore, using the fact that m is spanned by D and 
E, it is easy to verify that for an arbitrary matrix U m ° G trig we have 
[U m 'o,V m> ] =sF, where 

/ 1 o\ 

-10 

1 

\ 0-10/ 



F:= 



and s G R. After choosing the vectors t/ 3 , U m o , V m , the real numbers t and 
s are uniquely determined. Define W := [C/ s , V s } = 



( 



\ 



) 



yix 2 - 2/2 xi 2/1 x 3 - 2/3 xi 2/1 x 4 - y 4 x\ 

2/2 xi -y\x% y 2 x 3 - y 3 x 2 y 2 x A - y 4 x 2 

2/3 xi - 2/1 x 3 y 3 x 2 -y 2 x 3 y 3 x 4 - y 4 2:3 

\ 2/4 xi - 2/1 a?4 2/4 £ 2 - 2/2 2?4 2/4 £3 - 2/3 ^4 

We will finish the proof by showing that there is a constant C > such that 

\[u,vf\ 2 >c 2 \[u m ',v m '} m '\ 2 . 

Using the Euclidean scalar product given by (A, B) = ti{A t B) and the 
corresponding norm on R 4x4 , we have 



\U m , V n 



\tE + sF\ 



4(s 2 + t 2 ) 



Assume that y = (2/1, 2/2,2/3,2/4) € R 4 is a fixed unit vector. We choose the 
orthonormal basis 

2/, ei := (-2/4,-2/3,2/2,2/1), e 2 := (-2/2,2/1,-2/4,2/3), e 3 := (2/3,-2/4,-2/1,2/2) 

of R 4 , and write x = ao2/ + aiei + 0262 + 0363 with a,j G R. The squared 
length of W is tr((ya; f — xy t ) t (yx t — xy t )) = 2 |x| |y| — 2(x,y) = 2(a 2 +a 2 +a1). 
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The length of the orthogonal projection of W on the linear subspace spanned 
by E is 

\y\x A -y A xi + 2/2X3 -V3X 2 \ = |(-jt/4, -2/3,2/2, 2/i)(xi,x 2 ,X3,x 4 ) t | = |ai| . 

The length of the orthogonal projection of W on the linear subspace spanned 
by F is 

\VlX2 -y 2 X\ +2/3X4 -2/4X3! = |(-2/2,2/l,-2/4,2/3)(xi,X2,X3,X4) t | = |<X 2 | • 

We have, since \sE\ = 2s and \tF\ = 2t, 



2 



[U, VY = a { + ai + 2a\ + (ai ± 2s)^ + (o 2 ± 2t) 



2 



= 2a? + 2a 2 , + 2a| ± 4ais ± Aa 2 t + 4s s + 4t 2 
= 2a§ + 2 (ai ± sf + 2 (a 2 ± t) 2 + 2s 2 + 2t 2 

> 2(s 2 + t 2 ) = ^ \sE + tF\ 2 = ]- I [U m ',V m '] m 

This finishes the proof. □ 

6. Regular subgroups 

A closed subgroup K of a compact Lie group G is called a regular subgroup 
if rktCcfX)) = rk(G) - rk(iT) + rk{Z(K)), where C G (K) is the centralizer 
of K in G and Z(K) is the center of X. In this case, we also call the Lie 
algebra t of K a regular subalgebra of the Lie algebra g of C When K Q G 
is a regular subgroup, there exists a maximal torus T of G with Lie algebra i 
such that i is spanned by a subset of i and the real root spaces g^, a £ S, 
where S is a symmetric and closed subsystem of the root system R of G. 
Consider a chain of compact Lie groups H C K C G where if is a regular 
subgroup of G. Then ii is regular subgroup of G if and only if it is a regular 
subgroup of K. 

Lemma 6.1. Let H C if C G be compact Lie groups such that G is simple 
and H, K are regular subgroups. Let T\ be a maximal torus of Ck{H). 
Assume that (K,H ■ Ti) is not a symmetric pair. Then there exist elements 
X, Y G p such that [X, Y] = and [X m , Y m ] m / 0. 

Proof. Let T 2 be a maximal torus of Cg{K). Let iJ' := H ■ T\ ■ T 2 , let 
K' := K -T 2 , let G' := G. Then rk(ii / ) = ik(K') = rk(G'). Now apply 
Corollary gH and Remark [42] to the chain H' C K' C G' . □ 

Proposition 6.2. Let K be a simple compact Lie group and let H C K be 
such that Ck{H) is of positive dimension and (K,H • Ti) is a symmetric 
pair for a maximal torus T\ C Ck(H). Then K/{H ■ Ti) is a Hermitian 
symmetric space and T\ is one- dimensional. 

Proof. This follows from the classification of symmetric spaces [H] . □ 
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Remark 6.3. Let H, K and T\ be as in Proposition [621 Let t = h©tiffimo be 
an orthogonal decomposition and let m = ti ©mo- (It follows that H C K is 
a regular subgroup |HJ.) Let To be a maximal torus of H. Then T := Tq ■ T\ 
is a maximal torus of K. Consider a root space decomposition of t and let 
Rk denote the set of roots. Let Rh denote the subsystem corresponding 
to the subgroup H. The set Rh consists of all roots in Rk which vanish 
on ti. Hence it follows that the projection of LYg, Yjg] on ti is non-zero for 
all (3 eR m := R K \ R H . 

Proposition 6.4. Let G be a compact Lie group. Let H C K C G be 
connected compact Lie groups such that H , K are regular subgroups. Assume 
that for each simple ideal L of g the condition i2. 3\) holds for the triple of 
Lie algebras (I n h, I H I, I). Then fOj) also holds for (H, K, G) . 

Proof. Let T 2 be a maximal torus of Cg(K). Let g = go © 0i © • • • Q n 
be a decomposition into ideals of g such that go is abelian and gi, . . . ,g n 
are simple. Define ti := Qi n t and fjj := Qi n I). Let rrij be the orthogonal 
complement of f}j in tj and let pj be the orthogonal complement of \)i in gj. 
Now assume that (t)i,h,Bi) satisfies (12. 3p with a positive constant Cj, i.e. 
we have 

|[i m ',r m f'| <Cj|[x,y]|. 

for i = l,...,n. Set C := max(Ci, . . . ,C n ). Now let X, Y S p, where 
m is the orthogonal complement of I) in £ and where p is the orthogonal 
complement of f) in g. Define i)' := \)q + . . . + t) n and let p' = po + ■ ■ ■ + p n 
be the orthogonal complement of \]' in g. Since t)' C h, it follows that p' D P 
and hence I,ye p'. Let £' := to + . . . + tn and let m' := mo + . . . + m n . We 
have m C m' + 1 2 . Since T 2 C C G (K), it follows that [X m , y m ] = LY m ', y m ']. 
We write X = Xq + . . . + X n and Y = Yq + . . . + Y n , where Xi,Y{ E Qi and 
compute 

n 

|[x m ,y m ] m | < |[x m ',y m '] m '| = \J2[x m \Y m ] m >\ < 

i=0 

n n 

<\Y J C i [X i ,Y i ]\<\Y,C[X i ,Y i ]\ = C\[X,Y]\. U 

i=i i=i 

We will now prove our classification result for chains of regular subgroups. 

Theorem 6.5. Let G be a compact Lie group. Let H C K C G be connected 
compact Lie groups such that H, K are regular subgroups of G. Lf the triple 
(H, K, G) satisfies condition h2.3\) then for each simple ideal g, of g one of 
the following is true. 

(1) g, n t = Qi, i.e. the simple ideal Qi is contained in t. 

(2) gj n t 7^ Qi and (Qi f] t, Qi H t)) is a symmetric pair, possibly such that 
Qi H t is contained in t). 

(3) gj = so(2n + 1), fl» D * = so(2n) and Qi n t) = su(n). 
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(4) Qi = sp(n) where each but one simple ideal of QiP\t is contained in f) 
and the one simple ideal not contained in \] is isomorphic to sp(l), 
standardly embedded. 

(5) Qi = Lie(G 2 ), Qi n t = so(4) and Qi n \] = su(2) such that QiHt) is 
contained in a subalgebra su(3) C Qi. 



Proof. Assume (12. 3p holds for the triple (H,K,G). Let To be a maximal 
torus of H. Let T\ be a maximal torus of Ck(H) and let T 2 be a maximal 
torus of Cg{K). Let R denote the root system of G with respect to T := 
To • T\ ■ T 2 . Generalizing the notation from Section [3j let Rh and Rk denote 
the closed symmetric subsets of R corresponding to the full rank subgroups 
H ■ T\ ■ T 2 C G and K ■ T 2 C G, respectively. Let R m = Rk \ Rh and 
R S = R\R K . 

Consider the decomposition K = Tk • K\ ■ . . . ■ K n where Tk is a torus 
and where the Ki are simple. Let G = Z ■ G\ ■ . . . ■ G m , where Z is the center 
of G and where G\, . . . , G m are the simple factors of G. Since K is a regular 
subgroup, there is a map /: {1, . . . , n} — > {1, . . . , m} such that ti C Qf^y 

Let M be the set of roots j3 G R m for which there is an i G {1, . . . , n} such 
that 0?- C ti, ti 7^ 0j(j) and (Ki,Ki n -ff) is not a symmetric pair. If the 
set £% is empty then one of the first two conditions in the statement of the 
theorem holds for each Qi. Thus we may assume the set £% is non-empty. 

If j3 £ M and g? C ti, then we may apply Lemma [6. II and Proposition 16.21 
to the chain f) fl ti C ti C 0/(j), showing that the pair (Ki,H H Ki) is as 
described in Proposition 16.21 

Since Gf^ is simple and ti 7^ 0/(i), it follows that for each (3 G Si there 
is at least one root a £ R s such that a + (3 £ R or a — (3 € R. Replacing 
f3 with — /3, if necessary, we may assume that a + (3 £ R. Let £P be the set 
of all pairs (a, (3) such that f3 £ &, a £ R s and a + (3 G -R. For each pair 
(a, f3) G ^, consider the set R(a,f3) of all linear combinations of a and (3 
with integer coefficients. This set R{a, f3) is a closed symmetric subsystem 
of R, thus the elements of R{a, f3) form a root system of rank two. Since 
(3 G R m , a,a + f3 G R s , it follows that the root system R(a,f3) is irreducible, 
thus of type A 2 , B 2 , or G 2 . 

First assume that among all elements of 3? there is at least one pair [a, (3) 
such that R(a,(3) is of type A 2 . Then we have R{a,f3) = {±a,±/3, ±(a + 
/?)}. Let 0' be the subalgebra of generated by the vectors X a , Y a , X@, 
Yp, X a+ p, Y a+ p. The algebra 0' is isomorphic to su(3). We may choose 
a three-dimensional complex representation of ; and assume that X a = 
E\2,Y a = iF 12 ,Xj3 = E 2 3,Yj3 = iF 23 ,X a+l3 = E 13 ,Y a+j3 = iF 13 . Define 
X m ' = E 23 , Y m ' = iF 23 , X s ' = E 12 + E 13 , Y s ' = i(F 12 - F 13 ). We see that 
[X m '+X s ',Y m ' + Y s '] = 0, while [X m ',Y m '\ = 2i{F 22 -F 33 ). By Remark E3 
this contradicts the assumption that the triple {H,K,G) satisfies (j2.3|) . 

Thus we may assume there is no pair {a, (3) G 3? such that R(a, (3) is of 
type A 2 . Hence each such R(a,/3) is of type B 2 or G 2 . 
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Assume (a,/3) £ & is such that R(a,/3) is of type G2. Since the root 
system G2 does not occur as a proper subsystem of any irreducible root 
system, it follows that the subalgebra g' generated by the vectors X\, Y\, 
A £ R(a,f3) is a simple ideal of g isomorphic to Lie(G2). It follows from 
Lemma 16.11 and Proposition 16.21 that the chain f) n 0' C t n g' C g' is as 
described in part ([5]) of the theorem, since otherwise we find a pair of com- 
muting vectors X,Y £ g', such that [X m ,Y m ] m / by the results of [KKJ 
Subsection 2.4]. 

Finally, we may assume that all root systems R{a,f3), {a, (3) £ ^, are of 
type B 2 . 

First assume that there is at least one pair (a, j3) £ S? such that R(a, f3) 
is of type B2 and such that j3 is a short root. Since i?x n i?(a, /3) is a closed 
symmetric subsystem of R(a,/3) it follows that ±/3 £ i? m , while A £ R s for 
all A £ R(a, f3) \ {±/3}. Let g' be the subalgebra of g generated by all vectors 
X\,Y\ where A £ R(a,/3) and let t' = t fl g'. The algebra g' is isomorphic 
to so(5) and t 1 is contained in the regular subalgebra so(3) ©so(2), it is 
isomorphic to either so (3) or 50 (3) ©so (2). We may choose a representation 
of g' on R 5 such that Eij with i = 1,2, j = 3, 4, 5 represent elements of s and 
£23, £45 £ t Define X m = E l2 and X s = -E 2i , Y m = E 13 and Y s = E u . 
Then [X,Y] = [E 12 -E 24 ,E l3 +E M ] = 0, yet [X m ,Y m ] = [E 12 ,E 13 ] = -E 23 . 
It follows from Remark 16.31 that the triple (H,K,G) does not satisfy (|2.3p . 

Now assume that for all root systems R{a,j3), (a,f3) £ 8? which are of 
type B2 each /3 is a long root. Then there are the following alternatives. 
Either among these there is at least one pair [a, /3) £ 3? such that there is 
a long root A £ R{a, /3) fl R s or all long roots in R(a, (3) are contained in 
Rk for each (a, (3) £ £? such that R(a,/3) is of type B2. 

Consider the first case. Let g' be the subalgebra of g generated by all 
vectors X\,Y\ where A £ R{a,j3) and let t' = t fl g' . The algebra g' is iso- 
morphic to so(5) and t' is one of the subalgebras su(2) or u(2) of g'. Choose 
a representation of g' on R 5 such that the subalgebra u(2) is represented 
by the linear combinations of the matrices E 23 ,E 2 4 + £"35, £"34 — E 2 §,E^. 
Define X = X m + X s , Y = Y m + Y 5 where X m = \(E 2b + E 34 ) and Y m = 
\{E 23 + £45). Take X s = E u + \(E 23 - £ 45 ) and Y s = E 12 + ±(£ 25 - E u ). 
Computing, we see that [X,Y] = while [X m ,Y m ] = -\{E 2A - E 35 ). It 
follows from Remark 16.31 that the triple {H, K, G) does not satisfy (|2.3|) . 

Finally we are left with the case where all root systems R{a,f3), {a,j3) £ 
& are of type B2, such that each /3 is a long root and all long roots in 
R{a,f3) are contained in Rk for each (a,/3) £ & . Fix one such root system 
R(oq, (3q). Let g' be the simple ideal of g which contains the vectors Xp ,Yp . 
Then g' is a regular subalgebra of g. We have to show that g' is as described 
in ([3]) or in @. Since R(oto,/3o) is of type B 2 , it follows that the simple 
factor of G corresponding to g' is of type B n , C„, or F4. 

Now the statement of the theorem follows from Lemma 16.61 Indeed, in 
case g' is of type B n , it follows from Lemma 16.61 that the simple ideal g' is 
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as in item ([3]) of the theorem. In case g' is of type F4, it follows from the 
Lemma that g' is as described in item ([2]) of the theorem. If g' is of type 
C n , we use the following counterexample to show that there can only be one 
simple ideal of t n g' which is not contained in f). Consider the chain of Lie 
algebras {0} C sp(l) 0sp(l) C sp(2). It follows from [Sj Lemma 2.2] that 
a pair of vectors X,Y with [X,Y] = 0, while [X m ,Y m ] m = [X m ,Y m ] ^ 
exists. Indeed, define e.g. 

j j + l\ „ ( -i-k i 



Then [X,Y] = 0, while [X m ,Y m ] m = [X m ,Y m ] ^ 0. Using Remark S21 it 
follows that g' is as described in item (j4|) of the theorem. □ 

To solve the remaining cases in the proof of the above theorem, we have 
to take a closer look at the root systems of type B n , C n and F4. 

Lemma 6.6. Let G be a simple compact Lie group of type B n , C n , or F4. Let 
H C K C G be connected subgroups such that rk(H) = rk(if) = rk(G) and 
such that (K, H) is a Hermitian symmetric pair. Then one of the following 
is true, where R s and R m are defined as in Section^ 

(1) There is pair of non- orthogonal roots (a, /3) S R s x i? m of the same 
length. 

(2) There is pair of non- orthogonal roots (a, j3) € R s x i? m such that a 
is a long root and (3 is a short root. 

(3) The triple of Lie algebras (fy,t, g) is isomorphic to 

(u(n),so(2n),so(2n + l)). 

(4) The triple (t),t, g) is of the following form: g = Sp(n) and for each of 
the simple factors ti,...,tf of t we have either tj C f) or tj = sp(l) 
and fynfj^u(l). 

(5) The triple (f),t,g) is isomorphic to (so(8),so(9),Lie(F4)). 

Proof. Assume first G is of type B n , i.e. g = so(2n + 1). The Lie algebra 
t is contained in a maximal subalgebra £' of full rank of g. This maximal 
subalgebra £' is conjugate to so(2^) ®5o(2n — 2^ + 1), where £ = 1, . . . ,n, [Ol 
Thm. 16, §3]. Using the notation of ^BtDl Prop. 6.5, Ch. V], the long roots 
of G are given by ±$ M ± fty where 1 < fi < v < n, while the short roots are 
±$j, where 1 < v < n. Then we may assume 

R K > = {±?V ±$ v \l<ix<v<lV £<fi<u<ri}U {±$„ I t < v < n}. 

and we have for the roots R s i C R s , corresponding to the orthogonal com- 
plement 5' of t' in g, 

R B i = {±^ ±-d v \l<ii<l<v<n}\J {±ti„ I 1 < v < £}■ 

We may assume £ < n since otherwise ([3]) holds. Since H ^ K, there is an 
element j3 € R m C R K i . If (3 is a long root, say f3 = i-tf^ ± ^j^, then we may 
choose an element a = ±$ K ± $\ € i? 3 ' ^ -Rs where k = fj, or k = v and ([I]) 
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holds. If /3 is a short root ±#j, for some £ < v < n, then choose a = ~d\ + 1?„ 
to show that ([2]) holds. 

Now assume G is of type C n , i.e. g = sp(n), n > 3. The Lie algebra 6 
is again contained in maximal subalgebra t' of full rank of g. The maximal 
subalgebras t' of full rank in sp(n) are conjugate to u(n), or sp(£)(&sp(n — £), 
where £ = 1, . . . , |_§J, see [O, Thm. 16, §3]. It follows by induction that all 
simple factors of K are of type A& or C&, k < n. Now, using the notation 
from [BtDl Prop. 6.6, Ch. V], the short roots of G are ±$^ =L i9 v where 
1 < /i < v < n and the long roots are ±2$^ where 1 < v < n. Assume first t 
is contained in a maximal subalgebra of g conjugate to u(n). Then we may 
assume 

R K > = {d^ -•d v \l<v,n<n, n^v}- 

and we have for the roots R s i C R Sl corresponding to the orthogonal com- 
plement s' of t' in g, 

R s , = {±(^ + ■&„) | 1 < n < v < n,} U {±2^ | 1 < v < n}. 

Since H ^ K, there is an element )3 € i? m C i?^-/. Obviously, there is some 
element a € R$i which is not orthogonal to /3. Since Rk> contains only short 
roots, either ([T]) or ([2]) holds. 

Assume now t is not conjugate to a subalgebra of u(n), hence contained 
in a maximal subalgebra of g conjugate to sp(£)©sp(n — £). We may assume 

R K i = {±tf M ±i9 I/ |l<^<^<^v£<^<i/<n}U {d=2i? I/ | 1 < z/ < n}. 

and we have for the roots R s i C i? s , corresponding to the orthogonal com- 
plement s' of 6' in g, 

R s > = {±^ M ± ■&„ | 1 < n < £ < v < n}. 

Assume that @ does not hold. Then there is a simple ideal tj of fi which is 
either isomorphic to u(q) for some q G {1, . . . , n — 1} or isomorphic to some 
sp(g) for some q € {2, . . . , n — 1} and such that tj fl f) 7^ &,-. It follows from 
[Q| §3] that then i? m contains a short root. But for any root 7 the set R s > 
contains a (short) root which is not orthogonal to 7. Thus (H|) holds. 

Now assume G is of type F4. The long roots are ±1?^ ±$„, 1<//<z/<4 
and the short roots are given by either ±19^, 1 < tt < 4 or f^^ 1 ^ ^2 i 
t?3 ± #4), see [B] or [U]. The maximal subgroups of maximal rank in F4 are 
Spin(9), Sp(3) • Sp(l), and SU(3) • SU(3), see P Thm. 16, §3]. 

Assume first that K is conjugate to a subgroup of SU(3) • SU(3). We 
deduce from [01 Thm. 16, §3] that the root system corresponding to this 
subgroup consists of six short roots and six long roots. The 24 long roots 
from R comprise a subsystem of type D4. Thus for any element 7 in R there 
is a long root in R s i which is non-orthogonal to 7. Hence ([2]) holds. 

Now assume K is conjugate to a subalgebra of Sp(3) • Sp(l). The root 
system of this group consists of 12 short and 8 long roots. These 8 long 
roots correspond to a subalgebra of type 4 • sp(l) = so(4) © so(4) C 50(8). 
A similar argument as above shows that ([2]) holds. 
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Finally, assume K is conjugate to a subgroup of Spin(9). We have 
R K < = {±tf M ±^|1</U<^<4}U {±1^ | 1 < n < 4}. 

If ET is any maximal connected subgroup of maximal rank in Spin (9) then 
((2|) holds, cf. [0, Thm. 16, §3]. Thus we may assume K = Spin(9). Note 
that for every root 7 in R there is a (short) root in the set R s i = {^(±$1 ± 
t?2±??3± $4)} which is not orthogonal to 7. Thus if (HJ does not hold then 
R m consists exclusively of long roots. It follows that H = Spin (8). □ 

Remark 6.7. For items (1), (2) (3) and (5) in Theorem 16.51 we know that 
condition (|2.3p holds for the chains (f) C\Qi,tC\ Qi,Qi). However, we con- 
jecture that condition f|2.3|) holds also for each chain of regular subgroups 
(H,K,G) = (Sp(l)"- 1 ,Sp(l)",Sp(n)) with n > 2. If the conjecture is true, 
it follows from Proposition 16.41 that the statement in Theorem 16.51 can be 
improved to "if and only if" . To our knowledge, there are no known exam- 
ples of chains (H, K, G) satisfying condition (|2.3p which contain non-regular 
subgroups, cf. |KK| . 
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